Machine Learning

• Calculate the cross entropy of the prediction with respect to the values that are expected.

• Rectified Linear Unit

\[ \mathrm{ReLU}(x) := \max (0,x). \]

\[ \mathrm{LReLU}(x) := \max(x, \varepsilon x) \] where \( 0 < \varepsilon \ll 1. \)

• Exponential Linear Unit

\[ \mathrm{ELU}(x) := \begin{cases} x & x >0, \\ \alpha(e^x -1) & x\le 0 \] where \( \alpha \ge 0 \).

• Researched in 80s and 90s.
• But what is a neural network? | Deep learning chapter 1 - YouTube
• It is useful to have rough concrete meaning of each layer (beforehand), to assess the network.
• Layer \(n\) Neurons \[a_{j}^{(n)}=\sigma\left( b_{j}^{(n)} + \sum_{i}w_{ij}^{(n)}a_{i}^{(n-1)}\right)\]
  • where \(\sigma: \mathbb{R} \to [0,1]\) is usually the sigmoid function, aka logistic curve: \[ \sigma(x) = \frac{1}{1+e^{-x}} \]
    • It could be any other functions, such as ReLU.
  • with matrix notation: \[ \mathbf{a}^{(n)} = \sigma\left(\mathbf{b}^{(n)}+\mathbf{W}^{(n)}\mathbf{a}^{(n-1)}\right) \]
• The entire neural network can be thought of as a function, as well as each neuron. \[ \mathbf{a}^{(-1)} = \mathbf{M}\left(\mathbf{\mathbf{a}}^{(0)}\right) = \dots\mathbf{M}^{(2)}\mathbf{M}^{(1)}\mathbf{a}^{(0)} \]

• RNN
• Serial processing. Like for audio.

• Used for image processing
• Perform multiple 2D convolution at each layer, to detect features.

• Extract the core features and lift back to the original shape.

• Self Attention Layer: Dynamic layer that adapt the weights based on the context. Measures the distance in the concept space.
• It is so named because it transforms the meaning of a sentence in the concept space.
• It is shallower but wider.
• Transformer Neural Networks Derived from Scratch - YouTube
• Calculate the attention of every pair of input vectors with an attention head neural network, and use it as a weight to reduce it down.

• Use the output, as the input for the next inference.

When generating an image, the diffusion model performs Langevin sampling, specifically using the Euler-Maruyama method: \[ \mathbf{x}_{t+1} = \mathbf{x}_t + \frac{\epsilon}{2} F(\mathbf{x}_t) + \sqrt{2\epsilon} \mathbf{z}_t, \] in order to sample from the space of all images, where the score function \( F \) is approximated by the trained model \( M_{\theta} \).

The random noise \( \mathbf{z}_t \) serves two main purpose.

• Escaping the local maxima,
• Providing high frequency information.

The model only learns to remove high frequency data, since they have to remove noise. Therefore constant supply of high frequency data is crucial for the generation of realistic image. Put it in other words, the noise alleviates the high correlation of contiguous pixels, generating a more realistic image.

• Move in the direction of \(-\nabla C(\theta)\), repeatedly.
• In order to calculate the gradient, the activation must be continuous and differentiable.

• Method of simplifying calculation by calculating layer by layer, starting from the last layer.
• \[ \frac{\partial C_i}{\partial a_{k}^{(L-1)}} = \sum_j \frac{\partial C_i}{\partial a_j^{(L)}}\frac{\partial a_j^{(L)}}{\partial a_k^{(L-1)}} \]
• Note that the gradient is readily calculated at each layer, since you already have all the values that the gradients is evaluated at.
• This is an application of ((65747a84-db71-4b38-aaf9-a357720b2aa9)), which enables multiple partial derivative at the same time.
• e.g. \[ \frac{\partial C_i}{\partial a_{k}^{(L-1)}} = \sum_j \left[2(a_j^{(L)}-y_j)\right]\left[w_{jk}^{(L)}\sigma'\!\!\left( z^{(L)}_j\right)\right] \]
• where \(z^{(L)}_j = b_{j}^{(L)} + \sum_k w_{jk}^{(L)}a_{k}^{(L-1)}\).

• For any policy \(\pi\),

  \begin{align*} v_\pi(s) &= \sum_{a\in \mathcal{A}(s)}\pi(a|s)q_\pi(s,a)\\ q_\pi(s,a) &= \sum_{s'\in\mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)[r+\gamma v_\pi(s')]\\ \end{align*}
  • This is due to the fact \[ \mathrm{E}_\pi[G_{t+1}\mid s, a] = \mathrm{E}_\pi[v_\pi(S_{t+1})\mid s,a] \] by the law of total probability

• Consequently,

  \begin{align*} v_\pi(s) &= \sum_{a\in \mathcal{A}(s)}\pi(a | s)\sum_{s'\in \mathcal{S}, r\in \mathcal{R}} p(s', r |s, a)(r+\gamma v_\pi(s'))\\ q_\pi(s,a) &= \sum_{s'\in\mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)\left[r+\gamma \sum_{a'\in \mathcal{A}(s')}\pi(a'|s')q_\pi(s',a')\right] \end{align*}

• The goal is to find the optimal policy \(\pi_*\) which satisfies:
  • \[ \forall \pi, \forall s \in \mathcal{S}, v_*(s) := v_{\pi_*}(s) \ge v_\pi(s), \]
  • \[ \forall \pi, \forall s\in \mathcal{S}, \forall a \in \mathcal{A}(s), q_*(s,a) := q_{\pi_*}(s,a) \ge q_\pi(s,a). \]
  • The existence of such policy is guaranteed.

• Since the model is unknown, estimating the action value is more feasible.

• The optimal deterministic policy \(\pi_*\): \[ \pi_*(s) = \operatorname*{argmax}_a q_*(s,a), \]
• Therefore, the approximation of the optimal policy is chosen greedily: \[ \pi'(s) = \operatorname*{argmax}_a q_\pi(s,a). \]

• On Policy TD Control
• The temporal difference learning is used. And the updates are applied during the epoisodes.
• "Sarsa" comes from 1-step Sarsa, in which every update operates on \(s_t^m, a_t^m, r_{t+1}^m, s_{t+1}^m, a_{t+1}^m\).

• n-Step Q-Learning

If an agent only visits a minuscule subset of all states, how should it generalize that experience to select high reward actions over the remaining states?

• Supervised Learning
  • Learn from noisy samples.
  • SL applied for RL is called function approximation.

• When \(\mathcal{S}\) is large, the degree of freedom is reduced by assuming: \[ v_\pi(s) \approx \hat{v}(s, \mathbf{w}) \]
• where \(\mathbf{w}\in \mathbb{R}^d\).
  • Since a state \(s\) is arbitrary there must exists features \(x_i : \mathcal{S}\to \mathbb{R}^n\) that describes the state.

• Assumption
  • Action space \(\mathcal{s}\) is not large.
  • Episodic case. otherwise it is different
• The state value function \(\hat{v}(s, \mathbf{w})\) is now action value function \(\hat{q}(s,a,\mathbf{w})\).

• Monte Carlo algorithm
  • Update happens after a full episode is performed.
• Assume
  • Policy \(\pi(a|s, \bm{\theta})\) is differentiable
• Initial setup
  • Functional form of \(\pi\)
  • Initial \(\bm{\theta}\)
  • Step size \(\alpha\)
• Softmax function is often used to unconstrain the parameter \(\theta_i\).
  • And the policy at one point is the average of the parameters weighted by the normalized radial basis, and then applying softmax to it.
  • It is not exactly linear but good enough.
• Update Rule
  • \[ \bm{\theta} \leftarrow \bm{\theta} + \alpha g_t^m \nabla\ln\pi(a_t^m|s_t^m, \bm{\theta}) \]
  • This reinforces the probability of taking action \(a_t^m\) when in state \(s_t^m\) in proportion to the return \(g_t^m\).
  • \(\ln\) is there to compensate for the effect of selection frequency on descent.
    • The high selection rate is exactly canceled by dividing the step by the propability.

• If \(\forall g_t^m \gg 1\), the optimization will take quite long. because the reinforce algorithm will never learn to not do things, only do more good things.
  • Replace \(g_t^m\) with \(g_t^m - b(s_t^m)\), where \(b\) is called the baseline.
  • Baseline can be any function that does not depend on action and does not make the problem diverge.
  • Natural choice of the baseline is the state value, the expected return of that state:
    • \[ b(s_t^m) = \hat{v}(s_t^m, \mathbf{w}) \]
    • The agent would prefer the action that increases the expected return.

• The objective to optimize in the policy gradient method is
  • Episodic case: \[ v_{\pi_{\bm{\theta}}}(s_0) = \mathrm{E}_{\pi_{\bm{\theta}}}[G_0|S_0 = s_0] \]
• Theorem
  • \[ \nabla\big|^{\bm{\theta}} v_{\pi_{\bm{\theta}}}(s_0) \propto \sum_{s\in \mathcal{S}}\mu_{\pi_{\bm{\theta}}}(s)\sum_{a\in \mathcal{A}(s)}q_{\pi_{\bm{\theta}}}(s, a)\nabla\big|^{\bm{\theta}}\pi(a|s, \bm{\theta}) \]
    • The gradient of the state value of the initial state, i.e. the gradient of the total return is proportional to the action value weighted average of the gradient of policy at each state-action pair.

If an algorithm approximates this direction \(\nabla\big|^{\bm{\theta}}v_{\pi_{\bm{\theta}}}\), it'll approximately optimize the objective.

• PGM may be slow at converging because it takes smooth transition for it to converge, but because of that it is more likely to converge eventually without getting stuck.
• It solves only for the policy.
  • No detailed knowledge is gained.
  • It bypasses the complexity.
• It can learn stochastic policies.
  • The level of stochasticity can be learned!
  • On the other hand, \(\epsilon\)-greedy method only has a fixed stochasticity.
• It avoids computationally intensive argmax operations.

• Proximal Policy Optimization